Control method, control system, and smart glasses for first person view unmanned aerial vehicle flight

ABSTRACT

A system for controlling an unmanned aerial vehicle (UAV) includes smart glasses configured to establish a first channel directly with the UAV, receive first person view (FPV) image data directly from the UAV through the first channel, and display the FPV image data, a remote control configured to establish a second channel directly with the UAV and send a first flight control instruction corresponding to the FPV image data directly to the UAV through the second channel, and establish a fourth channel directly with the mobile terminal, and a mobile terminal configured to display the FPV image data received from the UAV and forwarded by the remote control.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.16/133,316, filed on Sep. 17, 2018, which is a continuation ofInternational Application No. PCT/CN2016/080605, filed on Apr. 29, 2016,the entire contents of all of which are incorporated herein byreference.

TECHNICAL FIELD

The present disclosure relates to the field of unmanned aerial vehicleand, more particularly, to a control method, a control system, and smartglasses for first person view flight of unmanned aerial vehicle.

BACKGROUND

In a first person view (FPV) flight, images captured by a camera of anunmanned aerial vehicle (UAV) are transmitted to glasses in real timethrough an image transmission circuit at the UAV. A user can watch theimages captured by the camera and control a flight of the UAV. Throughthe FPV flight, the user may not need to look up to the UAV, but canobtain the same view as the UAV, so as to control the UAV under the viewof the UAV. Thus, an immersive flight experience may be achieved.

In conventional technologies, for achieving an FPV flight, the UAV and aremote controller communicate with each other through wirelesscommunication means such as a private communication protocol or wirelessfidelity (WiFi). The remote controller receives image signals from theUAV and sends the image signals to the glasses through wiredcommunication means such as universal serial bus (USB). However, in thisway, because a remote controller is needed for relay, delays in imagetransmission signals and control signals may occur, and the UAV flightmay not be promptly and accurately controlled, resulting in safetyhazards.

SUMMARY

In accordance with the disclosure, there is provided a system forcontrolling an unmanned aerial vehicle (UAV) that includes smart glassesand a remote controller. The smart glasses are configured to establish afirst channel directly with the UAV, receive first person view (FPV)image data directly from the UAV through the first channel, and displaythe FPV image data. The remote controller is configured to establish asecond channel directly with the UAV, and send a flight controlinstruction directly to the UAV through the second channel.

Also in accordance with the disclosure, there is provided a method forcontrolling an unmanned aerial vehicle (UAV). The method includescontrolling smart glasses to establish a first channel directly with theUAV, controlling the smart glasses to receive first person view (FPV)image data directly from the UAV through the first channel and todisplay the FPV image data, controlling a remote controller to establisha second channel directly with the UAV, and controlling the remotecontroller to send a flight control instruction directly to the UAVthrough the second channel.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of exemplary smart glasses according tovarious disclosed embodiments of the present disclosure.

FIG. 2 is another schematic view of exemplary smart glasses according tovarious disclosed embodiments of the present disclosure.

FIG. 3 is a schematic view of an exemplary unmanned aerial vehicle (UAV)control system according to various disclosed embodiments of the presentdisclosure.

FIG. 4 is a schematic view of another exemplary UAV control systemaccording to various disclosed embodiments of the present disclosure.

FIG. 5 is a schematic view of another exemplary UAV control systemaccording to various disclosed embodiments of the present disclosure.

FIG. 6 is a flowchart of an exemplary control method for a first personview flight of unmanned aerial vehicle according to various disclosedembodiments of the present disclosure.

FIG. 7 is a flowchart of another exemplary control method for a firstperson view flight of unmanned aerial vehicle according to variousdisclosed embodiments of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Technical solutions of the present disclosure will be described withreference to the drawings. It will be appreciated that the describedembodiments are some rather than all of the embodiments of the presentdisclosure. Other embodiments conceived by those having ordinary skillsin the art on the basis of the described embodiments without inventiveefforts should fall within the scope of the present disclosure.

As used herein, when a first assembly is referred to as “fixed to” asecond assembly, it is intended that the first assembly may be directlyattached to the second assembly or may be indirectly attached to thesecond assembly via another assembly. When a first assembly is referredto as “connecting” to a second assembly, it is intended that the firstassembly may be directly connected to the second assembly or may beindirectly connected to the second assembly via a third assembly betweenthem. The terms “perpendicular,” “horizontal,” “left,” “right,” andsimilar expressions used herein are merely intended for description.

Unless otherwise defined, all the technical and scientific terms usedherein have the same or similar meanings as generally understood by oneof ordinary skill in the art. As described herein, the terms used in thespecification of the present disclosure are intended to describeexemplary embodiments, instead of limiting the present disclosure. Theterm “and/or” used herein includes any suitable combination of one ormore related items listed.

FIG. 1 is a schematic view of exemplary smart glasses according tovarious disclosed embodiments of the present disclosure. FIG. 2 isanother schematic view of other exemplary smart glasses according tovarious disclosed embodiments of the present disclosure. As shown inFIGS. 1 and 2, a set of glasses include a housing. The housing includesa first side 11, a second side 12, a third side 13, a fourth side 14, afirst lens group 15, a second lens group 16, a first display screen 17,a second display screen 18, a first optical mirror 19, a second opticalmirror 20, and a selective light-shielding component 21.

The first side 11 and the second side 12 are arranged opposite to eachother. The third side 13 and the fourth side 14 are both connected tothe first side 11 and the second side 12, and are opposite to eachother.

The first lens group 15 and the second lens group 16 are spaced apartfrom each other, and are arranged at the first side 11.

The first display screen 17 and the second display screen 18 arearranged at the third side 13 and the fourth side 14, respectively.

The first optical mirror 19 and the second optical mirror 20 arearranged crosswise in the housing. The first optical mirror 19 isconfigured to reflect image light outputted by the first display screen17 to the first lens group 15. The second optical mirror 20 isconfigured to reflect image light outputted by the second display screen18 to the second lens group 16.

The selective light-shielding component 21 is arranged at the secondside 12. The selective light-shielding element 21 can be switchedbetween a light-shielding status and a light-transmitting status. Insome embodiments, the first optical mirror 19 and the second opticalmirror 20 may include semi-reflective semi-transparent (SRST) mirrors.When the selective light-shielding component 21 is at alight-transmitting status, the first optical mirror 19 and the secondoptical mirror 20 may further transmit light of a real scene outside thesmart glasses to the first lens group 15 and the second lens group 16,such that the user can observe the real scene and a virtual scenethrough the first lens group 15 and the second lens group 16 at the sametime.

In some embodiments, the first display screen 17 and the second displayscreen 18 may have, for example, a resolution of, e.g., 1920×1080 or2560×1440, and a refresh rate of, e.g., 60 Hz. Further, the firstdisplay screen 17 and the second display screen 18 may be arranged overtwo ends of the smart glasses, respectively, i.e., the third and fourthsides of the housing of the smart glasses, and may be parallel to adirection of an optical axis of a user eye.

In some embodiments, each of the first lens group 15 and the second lensgroup 16 may include a combination of a concave lens and a convex lens.The concave lens may reduce a field of view and allow light within acertain range to pass through a lens group. The convex lens may furtherenlarge an image to enhance a sense of immersion. Each of the first lensgroup 15 and the second lens group 16 may be arranged at an angle withrespect to each of the first display screen 17 and the second displayscreen 18. In some embodiments, the angle may be, for example,approximately 90 degrees.

In some embodiments, optical axis directions of the first lens group 15and the second lens group 16 may be parallel to each other, and displaysurfaces of the first display screen 17 and the second display screen 18may be parallel to the optical axis directions of the first lens group15 and the second lens group 16.

In some embodiments, positions of the first lens group 15 and the secondlens group 16 at the first side 11 may be adjustable. For example, arelative movement between he first lens group 15 and the second lensgroup 16 at the first side 11 may be performed.

In some embodiments, the first optical mirror 19 and the second opticalmirror 20 may include glass or plexiglass that can partially-reflect andpartially transmit incident light. For example, light transmittance maybe selected to be approximately 30%, and light reflectance may beselected to be approximately 70%. In some other embodiments, the firstoptical mirror 19 and the second optical mirror 20 may both includelight-transmitting glass coated with an SRST film. The first opticalmirror 19 and the second optical mirror 20 may be arranged tilted withrespect to the optical axis directions of the first lens group 15 andthe second lens group 16, respectively. Tilt angles of the first opticalmirror 19 and the second optical mirror 20 with respect to the firstlens group 15 and the second lens group 16 can be adjusted according topositions of the first lens group 15 and the second lens group 16 at thefirst side 11. In some embodiments, the first optical mirror 19 and thesecond optical mirror 20 may be arranged tilted with respect to thefirst lens group 15 and the second lens group 16 at approximately 45degrees, respectively.

In some embodiments, the first optical mirror 19 and the second opticalmirror 20 may include, but are not limited to, reflective mirrors.

The light transmittance of the selective light-shielding component 21may be adjustable. In some embodiments, the selective light-shieldingcomponent 21 may include a liquid crystal display screen (LCD screen),and the LCD screen may adjust light transmittance according to anapplied voltage value. As such, the selective light-shielding component21 can operate in one of two statuses, i.e., a transparent status and alight-shielding status. In the transparent status, the lighttransmittance may be close to approximately 100%. In the light-shieldingstatus, light may not pass through the selective light-shieldingcomponent 21.

When the selective light-shielding component 21 is in thelight-shielding status, light outside the smart glasses may not enterthe smart glasses, and light from the first display screen 17 and thesecond display screen 18 may be reflected by the first optical mirror 19and the second optical mirror 20 and enter the first lens group 15 andthe second lens group 16. Accordingly, the visual experience of the usereyes may be immersion in virtual reality. The selective light-shieldingcomponent 21 may be arranged at certain angle(s) with respect to thefirst display screen 17 and the second display screen 18. In someembodiments, the selective light-shielding component 21 may be arrangedat approximately 90 degrees with respect to the first display screen 17and the second display screen 18.

When the selective light-shielding component 21 is adjusted to betransparent, light outside the glasses can enter into the glasses. Thelight may pass through the first optical mirror 19 and the secondoptical mirror 20, and may be superimposed on reflected light reflectedby the first optical mirror 19 and the second optical mirror 20 togenerate an augmented reality effect.

The superposition effect of the external scene with the first displayscreen 17 and the second display screen 18 may depend on lighttransmittances of the first optical mirror 19 and the second opticalmirror 20, an intensity of external light, and brightnesses of the firstdisplay screen 17 and the second display screen 18. The brightnesses ofthe first display screen 17 and the second display screen 18 can beadjusted in response to the external light intensity to achieve a stablesuperposition effect in different scenarios. In some embodiments, theuser can manually adjust the brightnesses of the first display screen 17and the second display screen 18.

In some embodiments, the smart glasses can directly establish a firstchannel with the UAV without relaying by the remote controller, suchthat the smart glasses can receive FPV image data directly from the UAVthrough the first channel and display the FPV image data, and maydisplay the FPV image data on the first display screen 17 and the seconddisplay screen 18. Here, directly establishing a channel between twodevices means the there is no third device between the two devices torelay data, i.e., data can be sent from one of the two devices to theother one of the two devices without going through the third device.

In some embodiments, the smart glasses can also obtain flight statusdata of the UAV directly from the UAV. The flight status data of the UAVmay be further displayed on the first display screen 17 and the seconddisplay screen 18, to superimpose on the FPV image data displayed on thefirst display screen 17 and the second display screen 18, such that theuser can also see the flight status data of the UAV at the same timethrough the first lens group 15 and the second lens group 16. The flightstatus data of the UAV may include, but is not limited to, a location ofthe UAV, a direction of the UAV, remaining power, a flight path,obstacle information, flight altitude and flight speed, a noseorientation of the UAV, an orientation of a gimbal at the UAV, anoperation prompt, an FPV video, etc.

In some embodiments, the above-described virtual scene may include avirtual operation interface. Referring to FIGS. 1 and 2, in theseembodiments, the smart glasses may further include a camera 22 and aprocessor (not shown in FIGS. 1 and 2) for processing images from thecamera 22. The camera 22 is arranged at the housing and captures animage of a gesture operation, also referred to as a gesture operationimage, of the user through the camera 22. The processor may be arrangedin the housing and may be configured to process the gesture operationimage, determine coordinates of the gesture operation, and to comparewith an imaging position of the virtual operation interface, such that acontrol instruction corresponding to the gesture operation of the usermay be recognized and a corresponding operation may be performed.

That is, an operation interface may be displayed on the first displayscreen 17 and the second display screen 18, and may be reflected by thefirst optical mirror 19 and the second optical mirror 20 to form avirtual operation interface. The camera 22 may capture a gestureoperation image of the user. The gesture operation image may beprocessed by the processor 23 to determine coordinates of the gestureoperation. The processor may further compare coordinates of the gestureoperation with an imaging position of the virtual operation interface,such that a control instruction corresponding to the gesture operationof the user may be recognized and a corresponding operation may beperformed.

When the virtual operation interface includes a virtual operationinterface for glasses menu, correspondingly the control instructions mayinclude a control instruction for the smart glasses. When the virtualoperation interface includes a virtual operation interface for the UAVremote controller, correspondingly, the control instruction may includea control instruction for the UAV. Control instructions for the UAV mayinclude control instructions for the gimbal and/or control instructionsfor the UAV flight. The control instructions for the gimbal may beconfigured to control the gimbal of the UAV and an imaging apparatuscarried by the gimbal. The control instructions for the UAV flight maybe configured to control the UAV flight.

In some embodiments, the smart glasses may also directly recognize auser gesture motion to perform a corresponding operation. In theseembodiments, referring to FIGS. 1 and 2, the smart glasses include acamera 22, as well as a memory and a processor (not shown in FIGS. 1 and2). The camera 22 is arranged at the housing. The camera 22 may beconfigured to capture, i.e., photograph, a gesture motion of a user. Thememory may store a plurality of user gesture motion models and aplurality of control instructions corresponding to the plurality of usergesture motion models. The memory and the processor both may be arrangedin the housing. The processor may be electrically connected to thecamera 22 and the memory. The processor may be configured to generate agesture motion model according to the user gesture motion captured bythe camera 22 to compare with the plurality of user gesture motionmodels stored in the memory. In response to the generated gesture motionmodel being consistent with one of the plurality of user gesture motionmodels stored in the memory, a control instruction corresponding to theone of the plurality of user gesture motion models may be triggered.That is, in response to the generated gesture motion model beingconsistent with one of the plurality of user gesture motion modelsstored in the memory, a control instruction corresponding to thegenerated gesture motion model may be triggered.

The plurality of user gesture motion models stored in the memory may beobtained by photographing a plurality of user gesture motions throughthe camera and defining each user gesture motion model in advance. Thatis, by capturing, i.e. photographing, and recognizing a plurality ofuser gesture motions in advance, corresponding user gesture motionmodels may be generated. Further, a control instruction uniquelycorresponding to each user gesture motion model may be defined andstored in the memory.

The control instruction may include a gimbal control instruction forcontrolling a gimbal of the UAV, and/or a glasses control instructionfor controlling the smart glasses, and/or a flight control instructionfor controlling the UAV flight. The control instruction may includeanother control instruction that is not listed.

In some embodiments, based on the camera 22 at the smart glasses,contents displayed on the first display screen 17 and the second displayscreen 18 may also be customized according to various applicationscenarios. For example, a face image, a two-dimensional (2D) code image,or the like may be captured through the camera 22. Correspondingly, theface image, the two-dimensional (2D) code image, or the like may bedisplayed on the first display screen 17 and the second display screen18. The processor may perform face recognition or two-dimensional coderecognition, such that pedestrian detection, face recognition,two-dimensional code recognition, and other functions may be achieved.Functions that can be achieved are not limited to the above-describedexamples.

The smart glasses of the disclosure can further establish a secondchannel directly with the remote controller, such that the smart glassescan receive a glasses control instruction directly from the remotecontroller through the second channel and perform a correspondingoperation according to the glasses control instruction.

That is, the menu function of the smart glasses can be achieved throughbutton control of the remote controller. For example, display backlightbrightness, camera settings, photographing, video recording and the likethat can be achieved by the smart glasses menu operations can also beaccomplished through five-dimensional buttons, also referred to as“five-dimensional keys,” on the remote controller. When the menufunctions of the smart glasses are implemented through the remotecontroller, the remote controller may send a disabling instruction tothe smart glasses through a third channel to disable the menu functionsof the smart glasses. In some embodiments, control functions of thesmart glasses corresponding to the first flight control instructions andgimbal control instructions may be disabled. That is, when the smartglasses detect that the remote controller is connected, the menufunctions of the smart glasses may be disabled, and the smart glassesmay be controlled by glasses control instructions of the remotecontroller.

As compared to controlling menus using buttons of the smart glasses,directly controlling the smart glasses through the buttons of the remotecontroller to achieve the menu functions of the smart glasses realizes amore convenient user operation and eliminate the need to switch betweenbutton operations of the remote controller and button operations of thesmart glasses.

In some embodiments, the smart glasses can further establish a thirdchannel directly with a mobile terminal, and the smart glasses mayfurther send the FPV image data and/or flight status data received fromthe UAV to the mobile terminal through the third channel for display.

Further, the mobile terminal may generate a first flight controlinstruction and/or a gimbal control instruction according to a userinput, and may send the first flight control instruction and/or thegimbal control instruction to the smart glasses through the thirdchannel. Further, the smart glasses may send the first flight controlinstruction and/or the gimbal control instruction to the UAV through thefirst channel.

In some embodiments, the smart glasses may disable control functions ofthe smart glasses corresponding to the first flight control instructionand/or the gimbal control instruction after receiving the first flightcontrol instruction and/or the gimbal control instruction from themobile terminal. That is, in this case, the control instruction from themobile terminal may be used to achieve the corresponding controlfunction.

In some embodiments, the smart glasses and the mobile terminal can bothachieve the corresponding control functions. If there is controlinstruction(s) from one of the smart glasses or the mobile terminal, anoperation corresponding to the control instruction may be performeddirectly. If there are control instructions from the smart glasses andthe mobile terminal at the same time, a priority may be determinedaccording to a preset rule, and an operation corresponding to a controlinstruction with a higher priority may be performed. The preset rule maybe set as needed. For example, an instruction from the smart glasses maybe prioritized, or a control instruction from the mobile terminal may beprioritized. The setting of the preset rule is not restricted in thepresent disclosure.

In some embodiments, the UAV can communicate with the smart glassesthrough a wireless channel, to perform image transmission and to sendand receive control signals. In some other embodiments, the UAV cancommunicate with the mobile terminal through wireless WiFi, to performimage transmission and to send and receive control signals.

The above-described second channel and third channel may includewireless channel(s) or wired channel(s).

The above-described implementation approaches are merely forillustrative purposes and are not intended to limit the scope of thepresent disclosure. Various other implementation approaches may beadopted. In some implementation approaches, the smart glasses, theremote controller, and the mobile terminal may be connected to eachother through wireless connection(s) and/or wired connection(s), e.g., aUSB connection.

In some embodiments, the smart glasses can also obtain a video file ofthe UAV directly from the UAV through the first channel, and play backthe video file on the smart glasses.

FIG. 3 is a schematic view of an exemplary UAV control system accordingto various disclosed embodiments of the present disclosure. As shown inFIG. 3, the UAV control system includes a remote controller 11 and smartglasses 12. The smart glasses 12 can directly establish a first channelwith the UAV 100 without relaying by the remote controller 11, such thatthe smart glasses 12 can receive FPV image data directly from the UAVthrough the first channel and display the FPV image data. The remotecontroller 11 can directly establish a second channel with the UAV 100without relaying by the smart glasses 12, such that the remotecontroller can send a first flight control instruction directly to theUAV through the second channel.

In some embodiments, the first flight control instruction may include acontrol instruction for controlling the UAV flight. For example, thefirst flight control instruction may include a control instruction forcontrolling a flight status of the UAV, such as taking off, returning,flying back, forward, to left, and to right, throttling, turning left,turning right, etc.

In some embodiments, the smart glasses 12 can further receive flightstatus data directly from the UAV through the first channel and displaythe flight status data. The flight status data may include, but is notlimited to, a location of the UAV, a direction of the UAV, remainingpower, a flight path, obstacle information, flight altitude and flightspeed, a nose orientation of the UAV, an orientation of a gimbal at theUAV, etc.

In some embodiments, the smart glasses 12 may include smart glassesconfigured to realize both virtual reality and augmented reality. Withthe smart glasses of the present disclosure, the flight status data maybe displayed in a manner of augmented reality in superposition with theabove-described FPV image data. For example, the FPV image data and theflight status data may be displayed in a picture-in-picture manner.

In some embodiments, the smart glasses 12 may include, but are notlimited to, virtual reality glasses or augmented reality glasses.

In some embodiments, the smart glasses 12 may sense an attitude of thesmart glasses 12 through a built-in attitude sensor and generate agimbal control instruction. The smart glasses 12 may further send agimbal control instruction directly to the UAV through the firstchannel. The attitude sensor may include, for example, an inertialmeasurement unit (IMU) or the like.

That is, the attitude of the smart glasses 12 can be sensed by theattitude sensor of the smart glasses 12 and converted into acorresponding gimbal control instruction for controlling the gimbal atthe UAV and an imaging apparatus carried by the gimbal, such as acamera, a camera telescope, a remote camera, or a measuring instrument.In some embodiments, the gimbal control instruction may be used toachieve stabilization of the imaging apparatus, to adjust an attitude ofthe imaging apparatus, e.g., a tilt angle and a photographing directionof the imaging apparatus, for achieving high-quality shooting and/orphotographing, etc. For example, a camera of the imaging apparatuscarried by the gimbal may be adjusted downward by adjusting the smartglasses downward, and the camera of the imaging apparatus carried by thegimbal may be adjusted to the left by turning the smart glasses to theleft, and the examples are merely for illustrative purposes and are notintended to limit the scope of the present disclosure.

In some embodiments, the smart glasses 12 can sense a user gesture andgenerate a glasses control instruction. The smart glasses 12 may furtherperform a corresponding operation according to the glasses controlinstruction. The glasses control instruction may include an instructionfor setting the smart glasses, such as display screen backlightbrightness adjustment, a resolution of the smart glasses 12, or thelike, and may also include a control instruction for controlling theimaging apparatus of the UAV, such as camera settings, photographing,video recording, etc.

That is, the menu operation of the smart glasses can be replaced by thesmart glasses sensing user gestures, and the smart glasses menufunctions can be realized through gesture operations. In someembodiments, the smart glasses 12 may include a front camera. In anaugmented reality mode, an operation may be performed by the userclicking a finger to a position where a glasses icon is located. Forexample, in an augmented reality mode, operation menus of the smartglasses 12 may be displayed on an image plane. The user may click on acorresponding operation menu, and the camera of the smart glasses 12 maycapture images of the user's click operation, and identify coordinatesof a stable position of the user's finger in a certain time duration. Bycomparing the stable position of the user's finger with positions of themenus in the glasses, a glasses control instruction corresponding to theuser's gesture operation may be recognized and a corresponding operationmay be performed.

In some embodiments, the smart glasses 12 can sense a user gesture andgenerate a second flight control instruction. The smart glasses 12 canfurther send the second flight control instruction directly to the UAVthrough the first channel.

In some embodiments, the second flight control instruction may be sameas the first flight control instruction, and both first and secondflight control instructions may include a control instruction forcontrolling the UAV flight. For example, the second flight controlinstruction may include a control instruction for controlling a flightstatus of the UAV, such as taking off, returning, flying back, forward,to left, and to right, throttling, turning left, turning right, etc.Thus, flight control of the UAV may be realized by sensing the user'sgesture operation through the smart glasses. In some embodiments, inresponse to the UAV receiving the second flight control instruction fromthe smart glasses 12 without receiving a first flight controlinstruction from the remote controller 11, the UAV may control the UAVflight under the second control instruction. In some other embodiments,in response to the UAV receiving the first flight control instructionfrom the remote controller 11 without receiving the second flightcontrol instruction from the smart glasses 12, the UAV may control theUAV flight under the first control instruction. In response to receivingthe flight control instructions from the smart glasses 12 and the remotecontroller 11 at the same time, the UAV may perform a prioritydetermination according to a preset rule, and may control the UAV flightunder a control instruction having a higher priority. The preset rulemay be set in advance according to actual needs. In some embodiments,the control instruction from the smart glasses may be prioritized. Insome other embodiments, the control instruction from the remotecontroller may be prioritized.

In some embodiments, the smart glasses may include a camera and aprocessor. The camera may capture a user gesture operation image. Theprocessor may process the gesture operation image, determine coordinatesof the gesture operation, and compare the coordinates of the gestureoperation with an imaging position of a virtual operation interface forcontrolling UAV displayed over the smart glasses, such that the secondflight control instruction corresponding to the user's gesture operationmay be recognized, and a corresponding operation may be performed.

That is, the virtual operation interface may be displayed on the displayscreen of the smart glasses, and the image of the user's gestureoperation is captured by the camera. The processor may process the imageof the user's gesture operation to determine coordinates of the gestureoperation, and compare the coordinates of the gesture operation with animaging position of the virtual operation interface, such that thesecond flight control instruction corresponding to the user's gestureoperation may be recognized, and a corresponding control operation maybe performed.

In some embodiments, the smart glasses may include a camera, a memory,and a processor. The camera may be configured to capture a gesturemotion of a user. The memory may store a plurality of user gesturemotion models and a plurality of second flight control instructionscorresponding to the plurality of user gesture motion models. Theprocessor may be electrically connected to the camera and the memory.The processor may be configured to generate a gesture motion modelaccording to the user gesture motion captured by the camera to comparewith the plurality of user gesture motion models stored in the memory.In response to the generated gesture motion model being consistent withone of the plurality of user gesture motion models stored in the memory,a second flight control instruction corresponding to the one of theplurality of user gesture motion models may be triggered. That is, inresponse to the generated gesture motion model being consistent with oneof the plurality of user gesture motion models stored in the memory, asecond flight control instruction corresponding to the generated gesturemotion model may be triggered.

The plurality of user gesture motion models stored in the memory may beobtained by photographing a plurality of user gesture motions throughthe camera and defining each user gesture motion model in advance. Thatis, by capturing and recognizing a plurality of user gesture motions inadvance, corresponding user gesture motion models may be generatedseparately. Further, a second flight control instruction uniquelycorresponding to each user gesture motion model may be defined andstored in the memory.

In some embodiments, the UAV can communicate with the smart glassesthrough a wireless channel, to perform image transmission and to sendand receive control signals. In some other embodiments, the UAV cancommunicate with the mobile terminal through wireless WiFi, to performimage transmission and to send and receive control signals.

In some embodiments, a plurality of smart glasses may be added to theUAV control system to operate together as needed.

In some embodiments, the smart glasses not only can play real-timevideos taken by the UAV, but also can obtain a video file of UAV fromthe UAV through a wired or wireless channel and play back the video fileon the smart glasses.

Referring to FIG. 3, in some embodiments, the smart glasses 12 canfurther establish a third channel directly with the remote controller11, such that the smart glasses 12 can receive a glasses controlinstruction directly from the remote controller 11 through the thirdchannel, and perform a corresponding operation according to the glassescontrol instruction. That is, the menu function of the smart glasses 12can be achieved through button control of the remote controller 11. Forexample, display backlight brightness, camera settings, photographing,video recording and the like that can be achieved by the smart glassesmenu operations can also be accomplished through five-dimensionalbuttons on the remote controller 11. When the menu functions of thesmart glasses 12 are implemented through the remote controller 11, theremote controller 11 may send a disabling instruction to the smartglasses 12 through the third channel to disable the menu functions ofthe smart glasses 12. In some embodiments, control functions of thesmart glasses 12 corresponding to the first flight control instructionsand gimbal control instructions may be disabled. That is, when the smartglasses 12 detect that the remote controller 11 is connected, the menufunctions of the smart glasses 12 may be disabled, and the smart glasses12 may be controlled by glasses control instructions of the remotecontroller.

As compared to controlling menus using buttons of the smart glasses,directly controlling the smart glasses 12 through the buttons of theremote controller 11 to achieve the menu functions of the smart glassesrealizes a more convenient user operation and eliminates the need toswitch between button operations of the remote controller 11 and buttonoperations of the smart glasses 12.

Referring to FIG. 3, in some embodiments, the control system may furtherinclude a mobile terminal 13. The mobile terminal 13 and the remotecontroller 11 may directly establish a fourth channel between eachother. The remote controller 11 may further send FPV image data and/orflight status data received from the UAV to the mobile terminal 13through the fourth channel for display.

Further, the mobile terminal may generate a first flight controlinstruction and/or a gimbal control instruction according to user input,and send the first flight control instruction and/or the gimbal controlinstruction to the remote controller 11 through the fourth channel. Theremote controller 11 may send the first flight control instructionand/or the gimbal control instruction to the UAV through the secondchannel. Accordingly, the flight of the UAV and an imaging apparatuscarried by the UAV can be controlled on the mobile terminal 13.

The remote controller 11 may disable control functions of the remotecontroller 11 corresponding to the first flight control instructionand/or the gimbal control instruction after receiving the first flightcontrol instruction and/or the gimbal control instruction from themobile terminal 13. That is, in this case, the control instruction fromthe mobile terminal 13 may be used to achieve the corresponding controlfunction.

In some embodiments, the remote controller 11 and the mobile terminal 13both can achieve the corresponding control functions. If there iscontrol instruction(s) from one of the remote controller 11 or themobile terminal 13, an operation corresponding to the controlinstruction may be performed directly. If there are control instructionsfrom the remote controller 11 and the mobile terminal 13 at the sametime, a priority may be determined according to a preset rule, and anoperation corresponding to a control instruction with a higher prioritymay be performed. The preset rule may be set as needed. For example, acontrol instruction from the remote controller may be prioritized, or acontrol instruction from the mobile terminal may be prioritized, whichis not restricted in the present disclosure.

In some embodiments, the control system may include a mobile terminal13. FIG. 4 is a schematic view of another exemplary UAV control systemaccording to various disclosed embodiments of the present disclosure. Asshown in FIG. 4, a mobile terminal 13 directly establishes a fourthchannel with smart glasses 12. The smart glasses 12 may further send FPVimage data and/or flight status data received from the UAV to the mobileterminal 13 through the fourth channel for display.

In some embodiments, the mobile terminal 13 may further generate a firstflight control instruction and/or a gimbal control instruction accordingto user input, and may send the first flight control instruction and/orthe gimbal control instruction to the smart glasses 12 through thefourth channel. Further, the smart glasses 12 may send the first flightcontrol instruction and/or the gimbal control instruction to the UAVthrough the first channel.

The smart glasses 12 may disable control functions of the smart glassescorresponding to the first flight control instruction and/or the gimbalcontrol instruction after receiving the first flight control instructionand/or the gimbal control instruction from the mobile terminal 13. Thatis, in this case, the control instruction from the mobile terminal maybe used to achieve the corresponding control function.

In some embodiments, the smart glasses 12 and the mobile terminal 13both can achieve the corresponding control functions. If there iscontrol instruction(s) from one of the smart glasses or the mobileterminal, an operation corresponding to the control instruction may beperformed directly. If there are control instructions from the smartglasses and the mobile terminal at the same time, a priority may bedetermined according to a preset rule, and an operation corresponding toa control instruction with a higher priority may be performed. Thepreset rule may be set as needed. For example, an instruction from thesmart glasses may be prioritized, or a control instruction from themobile terminal may be prioritized. The setting of the preset rule isnot restricted in the present disclosure.

Based on the above-described control system of the disclosure, theflight of the UAV may be controlled through the remote controller 11, animmersive FPV flight may be experienced through the smart glasses 12,and transmitted images may be watched in real time through the mobileterminal. Under this mode, single-player or multi-player control of theUAV may be realized. For example, during a single-player control, theuser may control the flight of the UAV through the remote controller 11,and may experience an immersive FPV flight through the smart glasses 12.The user may perform menu controls of the smart glasses, such as menucontrols associated with display backlight brightness, camera setting,photographing, video recording and the like, through five-dimensionalbuttons on the remote controller. As another example, during atwo-player or multiple-player control, a first user may control theflight of the UAV through the remote controller 11, a second user maywear the smart glasses 12 to join or leave at any time. Camera setting,photographing, and video recording may be controlled by the first useror the second user. As another example, a first user may wear the smartglasses 12 and use the remote controller 11 to control the UAV. A seconduser may view image transmission through a mobile terminal application(APP). Camera setting, photographing, and video recording may becontrolled by the second user through the mobile terminal. The aboveexamples are merely for illustrative purposes and are not intended tolimit the scope of the present disclosure. Control approaches may beselected and adjusted according to various application scenarios.

In the above-described UAV control system consistent with thedisclosure, the UAV, the remote controller, the smart glasses, and themobile terminal may be connected to each other through a privatewireless communication protocol, such that direct communications betweeneach other may be realized without a need relaying.

The above-described implementation approaches are merely forillustrative purposes and are not intended to limit the scope of thepresent disclosure. Various other implementation approaches may beadopted. In some implementation approaches, the smart glasses, theremote controller, and the mobile terminal of the UAV control system maybe connected through a wireless connection and/or a wired connection,e.g., a USB connection.

In the UAV control system consistent with the disclosure, the smartglasses can establish a first channel directly with the UAV withoutrelaying by the remote controller, such that the smart glasses canreceive FPV image data directly from the UAV through the first channeland display the FPV image data. The remote controller can establish asecond channel directly with the UAV without relaying by the smartglasses, such that the remote controller can directly send a firstflight control instruction to the UAV through the second channel. Assuch, since the remote controller and the smart glasses both mayestablish communication channels directly with the UAV, relaying may notbe needed for transmission of images and control signals, and delayduring transmission of images and control signals may be suppressed.Control accuracy of the UAV and safety of UAV flight can be improved.

In some embodiments, the menu function of the smart glasses can berealized through button control of the remote controller, such that theuser may not need to frequently switch between remote controller buttonsand the smart glasses, and the user operation may be more convenient.

In some embodiments, the user can view the image transmission on themobile terminal, and can further control the UAV through the mobileterminal, such that multiple-player control of the UAV and FPV flightexperience may be realized at the same time.

FIG. 5 is a schematic view of another exemplary UAV control systemaccording to various disclosed embodiments of the present disclosure. Asshown in FIG. 5, the UAV control system includes a UAV 100 and smartglasses 110. The smart glasses 110 can establish a first channeldirectly with the UAV 100 without a need for relaying, such that thesmart glasses can receive FPV image data directly from the UAV throughthe first channel and display the FPV image data. The smart glasses 110can further sense a user gesture, generate a first flight controlinstruction, and send the first flight control instruction directly tothe UAV through the first channel.

In some embodiments, the first flight control instruction may include acontrol instruction for controlling a UAV flight. For example, the firstflight control instruction may include a control instruction forcontrolling a flight status of the UAV, such as taking off, returning,flying back, forward, to left, and to right, throttling, turning left,turning right, etc.

In some embodiments, the smart glasses may include a camera and aprocessor. The camera may capture a user gesture operation image. Theprocessor may process the gesture operation image, determine coordinatesof the gesture operation, and compare the coordinates of the gestureoperation with an imaging position of a virtual operation interface forcontrolling the UAV displayed on the smart glasses, such that a firstflight control instruction corresponding to the user's gesture operationmay be recognized, and a corresponding operation may be performed.

That is, by recognizing the gesture operation image, determiningcoordinates of the gesture operation, and comparing the coordinates ofthe gesture operation with the imaging position of the virtual operationinterface of the UAV, an operation button position of the virtualoperation interface corresponding to the gesture operation may bedetermined, and a flight control operation corresponding to theoperation button may be directly performed.

In some embodiments, the smart glasses may include a camera, a memory,and a processor. The camera may be configured to capture a gesturemotion of a user. The memory may store a plurality of user gesturemotion models and a plurality of first flight control instructionscorresponding to the plurality of user gesture motion models. Theprocessor may be electrically connected to the camera and the memory.The processor may be configured to generate a user gesture motion modelaccording to the user gesture motion captured by the camera to comparewith the plurality of user gesture motion models stored in the memory.In response to the generated gesture motion model being consistent withone of the plurality of user gesture motion models stored in the memory,a first flight control instruction corresponding to the one of theplurality of user gesture motion models may be triggered. That is, inresponse to the generated gesture motion model being consistent with oneof the plurality of user gesture motion models stored in the memory, afirst flight control instruction corresponding to the generated gesturemotion model may be triggered.

The plurality of user gesture motion models stored in the memory may beobtained by photographing a plurality of user gesture motions throughthe camera and defining each user gesture motion model in advance. Thatis, by capturing and recognizing a plurality of user gesture motions inadvance, corresponding user gesture motion models may be generated.Further, a first flight control instruction uniquely corresponding toeach user gesture motion model may be defined and stored in a memory.

In some embodiments, the smart glasses 110 can further receive flightstatus data directly from the UAV through the first channel and displaythe flight status data. The flight status data may include, but is notlimited to, a location of the UAV, a direction of the UAV, remainingpower, a flight path, obstacle information, flight altitude and flightspeed, a nose orientation of the UAV, an orientation of a gimbal at theUAV, etc.

In some embodiments, the smart glasses may include smart glassesconfigured to realize both virtual reality and augmented reality. In theaugmented reality mode of the smart glasses of the present disclosure,the flight status data may be displayed in superposition with theabove-described FPV image data. For example, the FPV image data and theflight status data may be displayed in a picture-in-picture manner.

In some embodiments, the smart glasses 110 may include, but are notlimited to, virtual reality glasses or augmented reality glasses.

In some embodiments, the smart glasses 110 may sense an attitude of thesmart glasses 110 through a built-in attitude sensor and generate agimbal control instruction. The smart glasses 110 may further send thegimbal control instruction directly to the UAV through the firstchannel.

That is, the attitude of the smart glasses 110 can be sensed by theattitude sensor of the smart glasses 110 and converted into acorresponding gimbal control instruction for controlling the gimbal atthe UAV and an imaging apparatus carried by the gimbal, such as acamera, a camera telescope, a remote camera, or a measuring instrument.In some embodiments, the gimbal control instruction may be used toachieve stabilization of the imaging apparatus, to adjust an attitude ofthe imaging apparatus, e.g., a tilt angle and a photographing directionof the imaging apparatus, for high-quality shooting and/orphotographing, etc. For example, a camera of the imaging apparatuscarried by the gimbal may be adjusted downward by adjusting the smartglasses downward, and the camera of the imaging apparatus carried by thegimbal may be adjusted to the left by turning the smart glasses to theleft, and the examples are merely for illustrative purposes and are notintended to limit the scope of the present disclosure.

In some embodiments, the smart glasses 110 can sense a user gesture andgenerate a glasses control instruction. The smart glasses 110 mayfurther perform a corresponding operation according to the glassescontrol instruction. The glasses control instruction may include aninstruction for setting the smart glasses, such as display screenbacklight brightness adjustment, a resolution of the smart glasses 110,or the like, and may also include a control instruction for controllingan imaging apparatus of the UAV, such as camera settings, photographing,video recording, etc.

That is, the menu operation of the smart glasses can be replaced by thesmart glasses sensing user gestures, and the smart glasses menufunctions can be realized through gesture operations. In someembodiments, the smart glasses 110 may include a front camera. In anaugmented reality mode, an operation may be performed by the user usinga finger to click on a position where a glasses icon is located. Forexample, in an augmented reality mode, operation menus of the smartglasses 110 may be displayed on an image plane. The user may click on acorresponding operation menu. The camera of the smart glasses 110 maycapture an image of the user's click operation and identify coordinatesof a stable position of the user's finger in a certain time duration. Bycomparing the stable position of the user's finger with positions of themenus in the smart glasses, a glasses control instruction correspondingto the user's gesture operation may be recognized and a correspondingoperation may be performed.

Referring to FIG. 5, the control system consistent with the disclosurefurther includes a mobile terminal 120. The mobile terminal 120 mayestablish a second channel directly with smart glasses 110. The smartglasses 110 may further send FPV image data and/or flight status datareceived from a UAV 100 to the mobile terminal 120 through the secondchannel for display.

Further, the mobile terminal 120 may further generate a first flightcontrol instruction and/or a gimbal control instruction according touser input, and may send the first flight control instruction and/or thegimbal control instruction to the smart glasses 110 through the secondchannel. Further, the smart glasses 110 may send the first flightcontrol instruction and/or the gimbal control instruction to the UAV 100through the first channel.

The smart glasses 110 may disable control functions of the smart glassescorresponding to the first flight control instruction and/or the gimbalcontrol instruction after receiving the first flight control instructionand/or the gimbal control instruction from the mobile terminal 120. Thatis, in this case, the control instruction from the mobile terminal maybe used to achieve the corresponding control function.

In some embodiments, the UAV 100 can communicate with the smart glasses110 through a wireless channel, to perform image transmission and tosend and receive control signals. In some other embodiments, the UAV cancommunicate with the mobile terminal 120 through wireless WiFi, toperform image transmission and to send and receive control signals.

In some embodiments, a plurality of smart glasses 110 may be added tothe UAV control system to operate together as needed.

In some embodiments, the smart glasses 110 not only can play real-timevideos taken by the UAV, but also can obtain a video file of the UAVfrom the UAV through a wired or wireless channel and play back the videofile on the smart glasses 110.

In the embodiments of the present disclosure, in the UAV control system,smart glasses may communicate directly with the UAV, and directlycontrol the UAV without a need for relaying. As such, delay duringtransmission of images control signals may be suppressed, a controlaccuracy of UAV may be improved, and safety of UAV flight may beimproved.

In some embodiments, with the mobile terminal 120, the user canconveniently view the image transmission, and can realize control of theUAV and the smart glasses through operations on the mobile terminal,thereby facilitating the user's operation.

In the above-described UAV control system consistent with thedisclosure, the UAV 100, the smart glasses 110, and the mobile terminal120 may be connected to each other through a private wirelesscommunication protocol, such that direct communications between eachother may be realized without a need for relaying.

The above-described implementation approaches are merely forillustrative purposes and are not intended to limit the scope of thepresent disclosure. Various other implementation approaches may beadopted. In some implementation approaches, the smart glasses 110 andthe mobile terminal 120 of the UAV control system may be connectedthrough a wireless connection and/or a wired connection, e.g., a USBconnection.

FIG. 6 is a flowchart of an exemplary control method for UAV FPV flightaccording to various disclosed embodiments of the present disclosure.The control method for the FPV UAV flight may be implemented based onsmart glasses or a control system consistent with the disclosure, suchas the smart glasses described above in connection with FIGS. 1-2 or thecontrol system described above in connection with FIGS. 3 and 4. Withreference to FIG. 6, the control method is described below.

At S101, the smart glasses are controlled to establish a first channeldirectly with the UAV, and to receive FPV image data directly from theUAV through the first channel and display the FPV image data.

In some embodiments, the smart glasses may be controlled to establish aconnection with the UAV through a wireless private communicationprotocol, to achieve direction communications between the smart glassesand the UAV without relaying by a remote controller.

In some other embodiments, the UAV can also communicate with the mobileterminal through wireless WiFi, to perform image transmission and tosend and receive control signals.

A plurality of smart glasses may be added to operate together as needed.

In some embodiments, the control method may further includes controllingthe smart glasses to further receive flight status data directly fromthe UAV through the first channel and display the flight status data.The flight status data may include, but is not limited to, a location ofthe UAV, a direction of the UAV, remaining power, a flight path,obstacle information, flight altitude and flight speed, a noseorientation of the UAV, an orientation of a gimbal at the UAV, etc.

In some embodiments, the smart glasses may include smart glassesconfigured to realize both virtual reality and augmented reality.Through the smart glasses of the present disclosure, the flight statusdata may be displayed in superposition with the above-described FPVimage data in a manner of augmented reality. For example, the FPV imagedata and the flight status data may be displayed in a picture-in-picturemanner.

In some embodiments, the smart glasses may include, but are not limitedto, virtual reality glasses or augmented reality glasses.

In some embodiments, the control method may further include controllingthe smart glasses to sense an attitude of the smart glasses through abuilt-in attitude sensor and generate a gimbal control instruction, andcontrolling the smart glasses to further send the gimbal controlinstruction directly to the UAV through a first channel. The attitudesensor may include, for example, an inertial measurement unit (IMU) orthe like.

That is, the attitude of the smart glasses can be sensed by the attitudesensor of the smart glasses and converted into a corresponding gimbalcontrol instruction for controlling the gimbal at the UAV and an imagingapparatus carried by the gimbal, such as a camera, a camera telescope, aremote camera, or a measuring instrument. In some embodiments, thegimbal control instruction may be used to achieve stabilization of theimaging apparatus, to adjust an attitude of the imaging apparatus, e.g.,a tilt angle and a photographing direction of the imaging apparatus, forachieving high-quality shooting and/or photographing, etc. For example,a camera of the imaging apparatus carried by the gimbal may be adjusteddownward by adjusting the smart glasses downward, and the camera of theimaging apparatus carried by the gimbal may be adjusted to the left byturning the smart glasses to the left. The examples are for illustrativepurposes and are not intended to limit the scope of the presentdisclosure.

In some embodiments, the control method consistent with the presentdisclosure may further include controlling the smart glasses to sense auser gesture and generate a glasses control instruction, and controllingthe smart glasses to further perform a corresponding operation accordingto the glasses control instruction. The glasses control instruction mayinclude an instruction for setting the smart glasses, such as displayscreen backlight brightness adjustment, a resolution of the smartglasses, or the like, and may also include a control instruction forcontrolling an imaging apparatus of the UAV, such as camera settings,photographing, video recording, etc.

That is, the menu operation of the smart glasses can be replaced by thesmart glasses sensing user gestures, and the smart glasses menufunctions can be realized through gesture operations. In someembodiments, the smart glasses may include a front camera. In anaugmented reality mode, an operation may be performed by the user usinga finger to click on a position where a glasses icon is located. Forexample, in an augmented reality mode, operation menus of the smartglasses may be displayed on an image plane. The user may click on acorresponding operation menu. The camera of the smart glasses maycapture an image of the user's click operation and identify coordinatesof a stable position of the user's finger in a certain time duration. Bycomparing the stable position of the user's finger with positions of themenus in the glasses, a glasses control instruction corresponding to theuser's gesture operation may be recognized and a corresponding operationmay be performed.

In some embodiments, the smart glasses not only can play real-timevideos taken by the UAV, but also can obtain a video file from the UAVthrough a wired or wireless channel and play back the video file on thesmart glasses.

At S102, the remote controller is controlled to establish a secondchannel directly with the UAV, and to send a first flight controlinstruction directly to the UAV through the second channel.

In some embodiments, the first flight control instruction may include acontrol instruction for controlling the UAV flight. For example, thefirst flight control instruction may include a control instruction forcontrolling a flight status of the UAV, such as taking off, returning,flying back, forward, to left, and to right, throttling, turning left,turning right, etc.

In some embodiments, the method may further include controlling thesmart glasses to sense a user gesture and generate a second flightcontrol instruction, and controlling the smart glasses to further sendthe second flight control instruction directly to the UAV through thefirst channel.

In some embodiments, the second flight control instruction may be sameas the first flight control instruction, and both first and secondflight control instructions may include a control instruction forcontrolling the UAV flight. For example, the first flight controlinstruction may include a control instruction for controlling a flightstatus of the UAV, such as taking off, returning, flying back, forward,to left, and to right, throttling, turning left, turning right, etc.That is, flight control of the UAV may be realized by sensing the user'sgesture operation through the smart glasses. In some embodiments, inresponse to the UAV receiving the second flight control instruction fromthe smart glasses without receiving the first flight control instructionfrom the remote controller, the UAV may control the UAV flight under thesecond control instruction. In some embodiments, in response to the UAVreceiving the first flight control instruction from the remotecontroller without receiving the second flight control instruction fromthe smart glasses, the UAV may control the UAV flight under the firstcontrol instruction. In response to receiving the flight controlinstructions from the smart glasses and the remote controller at thesame time, the UAV may perform a priority determination according to apreset rule, and may control the UAV flight under a control instructionhaving a higher priority. The preset rule may be set in advanceaccording to actual needs. In some embodiments, the control instructionfrom the smart glasses may be prioritized. In some other embodiments,the control instruction from the remote controller may be prioritized.

In some embodiments, the method may further include controlling thesmart glasses to establish a third channel directly with the remotecontroller, such that the smart glasses can receive a glasses controlinstruction directly from the remote controller through the thirdchannel, and can perform a corresponding operation according to theglasses control instruction. That is, the menu function of the smartglasses can be achieved through button control of the remote controller.For example, display backlight brightness, camera settings,photographing, video recording and the like that can be achieved by thesmart glasses menu operations can also be accomplished throughfive-dimensional buttons on the remote controller. When the menufunctions of the smart glasses are implemented through the remotecontroller, the remote controller may send a disabling instruction tothe smart glasses through the third channel to disable the menufunctions of the smart glasses. In some embodiments, control functionsof the smart glasses corresponding to the first flight controlinstructions and gimbal control instructions may be disabled. That is,when the smart glasses detect that the remote controller is connected,the menu functions of the smart glasses may be disabled, and the smartglasses may be controlled by glasses control instructions of the remotecontroller.

As compared to controlling menus using buttons of the smart glasses,directly controlling the smart glasses through the buttons of the remotecontroller to achieve the menu functions of the smart glasses realizes amore convenient user operation and eliminate the need to switch betweenbutton operations of the remote controller and button operations of thesmart glasses.

The numerals S101 and S102 are merely for distinguishing differentprocesses, and are not intended to limit the sequence in which theprocesses are performed. The sequence of the above-described processesS101 and S102 is not restricted. In some embodiments, the two processesmay be performed, for example, at the same time.

In some embodiments, the control method may further include controllingthe mobile terminal to establish a fourth channel directly with theremote controller, and controlling the remote controller to receive FPVimage data and/or flight status data from the UAV through the secondchannel and send the FPV image data and/or the flight status data to themobile terminal through the fourth channel for display.

When the mobile terminal establishes the fourth channel directly withthe remote controller, the mobile terminal may be controlled to furthergenerate a first flight control instruction and/or a gimbal controlinstruction according to user input and to send the first flight controlinstruction and/or the gimbal control instruction to the remotecontroller through the fourth channel. Further, the remote controllermay send the first flight control instruction and/or the gimbal controlinstruction to the UAV through the second channel.

The remote controller may disable control functions of the remotecontroller corresponding to the first flight control instruction and/orthe gimbal control instruction after receiving the first flight controlinstruction and/or the gimbal control instruction from the mobileterminal. That is, in this case, the control instruction from the mobileterminal may be used to achieve the corresponding control function. Insome embodiments, the remote controller and the mobile terminal both canachieve the corresponding control functions. If there is controlinstruction(s) from one of the remote controller or the mobile terminal,an operation corresponding to the control instruction may be performeddirectly. If there are control instructions from the remote controllerand the mobile terminal at the same time, a priority may be determinedaccording to a preset rule, and an operation corresponding to a controlinstruction with a higher priority may be performed. The preset rule maybe set as needed. For example, a control instruction from the remotecontroller may be prioritized, or a control instruction from the mobileterminal may be prioritized. The setting of the preset rule is notrestricted in the present disclosure.

In some embodiments, the mobile terminal may be controlled to establisha fourth channel directly with the smart glasses, and the smart glassesmay be controlled to further send the FPV image data and/or the flightstatus data received from the UAV to the mobile terminal through thefourth channel for display.

When the mobile terminal establishes a fourth channel with the smartglasses, the mobile terminal may be controlled to further generate afirst flight control instruction and/or a gimbal control instructionaccording to user input, and to send the first flight controlinstruction and/or the gimbal control instruction to the smart glassesthrough the fourth channel. Further, the smart glasses may send thefirst flight control instruction and/or the gimbal control instructionto the UAV through the first channel.

The above-described third channel and fourth channel may include awireless channel and/or a wired channel.

The smart glasses may disable control functions of the smart glassescorresponding to the first flight control instruction and/or the gimbalcontrol instruction after receiving the first flight control instructionand/or the gimbal control instruction from the mobile terminal. That is,in this case, the control instruction from the mobile terminal may beused to achieve the corresponding control function.

In some embodiments, the smart glasses and the mobile terminal can bothachieve the corresponding control function. If there is controlinstruction(s) from one of the smart glasses or the mobile terminal, anoperation corresponding to the control instruction may be performeddirectly. If there are control instructions from the smart glasses andthe mobile terminal at the same time, a priority may be determinedaccording to a preset rule, and an operation corresponding to a controlinstruction with a higher priority may be performed. The preset rule maybe set as needed. For example, a control instruction from the smartglasses may be prioritized, or a control instruction from the mobileterminal may be prioritized. The setting of the preset rule is notrestricted in the present disclosure.

In the above-described embodiments, the UAV, the remote controller, thesmart glasses, and the mobile terminal may be connected to each otherthrough a private wireless communication protocol, such that directcommunications between each other may be realized without a need forrelaying.

In some embodiments, the mobile terminal may also be controlled toestablish a wireless channel directly with the UAV. The mobile terminalmay receive an FPV image from the UAV through the wireless channel anddisplay the FPV image. The UAV may receive a flight control instructionfrom the mobile terminal through the wireless channel.

The above-described implementation approaches are merely forillustrative purposes and are not intended to limit the scope of thepresent disclosure. Various other implementation approaches may beadopted. In some implementation approaches, the smart glasses, theremote controller, and the mobile terminal may be connected through awireless connection and/or a wired connection, e.g., a USB connection.

In some embodiments, the control method may further include controllingthe smart glasses to obtain a video file of the UAV directly from theUAV through the first channel, and play back the video file on the smartglasses.

In some embodiments, a plurality of smart glasses may be added tooperate together as needed.

In the above-described control method and system for controlling UAV FPVflight consistent with disclosure, the smart glasses can directlyestablish a first channel with the UAV without relaying by the remotecontroller, such that the smart glasses can receive FPV image datadirectly from the UAV through the first channel and display the FPVimage data. The remote controller can directly establish a secondchannel with the UAV without relaying by the smart glasses, such thatthe remote controller can send a first flight control instructiondirectly to the UAV through the second channel. As such, since theremote controller and the smart glasses both may establish communicationchannels directly with the UAV, relaying may not be needed fortransmission of images control signals, and delay during transmission ofimages control signals may be suppressed. A control accuracy of the UAVand safety of UAV flight may be improved.

In addition, the menu function of the smart glasses can be realizedthrough button control of the remote controller, such that the user maynot need to frequently switch between remote controller buttons and thesmart glasses, and the user operation may be more convenient.

In addition, the user can view the image transmission on the mobileterminal, and can further control the UAV through the mobile terminal,such that multiple-player control of the UAV and FPV flight experiencemay be realized at the same time.

FIG. 7 is a flowchart of another exemplary control method for UAV FPVflight according to various disclosed embodiments of the presentdisclosure. The control method for the FPV UAV flight may be implementedbased on smart glasses or a control system consistent with thedisclosure, such as the smart glasses described above in connection withFIGS. 1-2 and the control system described above in connection with FIG.5. With reference to FIG. 7, the control method is described below.

At S201, the smart glasses are controlled to establish a first channeldirectly with the UAV without a need for relaying, such that the smartglasses receive FPV image data directly from the UAV through the firstchannel and display the FPV image data.

In some embodiments, the control method may further include controllingthe smart glasses to further receive flight status data directly fromthe UAV through the first channel and display the flight status data.The flight status data may include, but is not limited to, a location ofthe UAV, a direction of the UAV, remaining power, a flight path,obstacle information, flight altitude and flight speed, a noseorientation of the UAV, an orientation of a gimbal at the UAV, etc.

In some embodiments, the smart glasses may include smart glassesconfigured to realize both virtual reality and augmented reality, andcan display the flight status data in a manner of augmented reality insuperposition with the FPV image data. For example, the FPV image dataand the flight status data may be displayed in a picture-in-picturemanner.

In some embodiments, the smart glasses may also include, but are notlimited to, virtual reality glasses or augmented reality glasses.

At S202, the smart glasses may be controlled to further sense a usergesture and generate a first flight control instruction, and to send thefirst flight control instruction directly to the UAV through the firstchannel.

In some embodiments, the first flight control instruction may include acontrol instruction for controlling the UAV flight. For example, thefirst flight control instruction may include a control instruction forcontrolling a flight status of the UAV, such as taking off, returning,flying back, forward, to left, and to right, throttling, turning left,turning right, etc.

Various implementation approaches may be used to sense the user gestureand generate the first flight control instruction, including twoexemplary approaches described below.

In one approach, the camera of the smart glasses may be controlled tocapture a user gesture operation image, and the gesture operation imagemay be processed to determine coordinates of the gesture operation. Thecoordinates of the gesture operation may be compared with an imagingposition of a virtual operation interface for controlling the UAVdisplayed on the smart glasses, such that the first flight controlinstruction corresponding to the user's gesture operation may berecognized, and a corresponding operation may be performed.

That is, by recognizing the gesture operation image, determiningcoordinates of the gesture operation, and comparing the coordinates ofthe gesture operation with the imaging position of the virtual operationinterface of the UAV, an operation button position of the virtualoperation interface corresponding to the gesture operation may bedetermined, and a flight control operation corresponding to theoperation button may be directly performed.

In another approach, the camera of the smart glasses may be controlledto capture a user gesture motion, and a gesture motion model may begenerated according to the user gesture motion, and be compared with aplurality of pre-stored user gesture motion models in the smart glasses.In response to the generated gesture motion model being consistent withone of the plurality of pre-stored user gesture motion models, a controlinstruction corresponding to the one of the plurality of user gesturemotion models may be triggered.

The plurality of pre-stored user gesture motion models may be obtainedby photographing a plurality of user gesture motions through the cameraof the smart glasses and defining each user gesture motion model inadvance. That is, by capturing and recognizing a plurality of usergesture motions in advance, corresponding user gesture motion models maybe generated. Further, a first flight control instruction uniquelycorresponding to each user gesture motion model may be defined andstored in a memory.

In some embodiments, the smart glasses may be controlled to sense anattitude of the smart glasses through a built-in attitude sensor andgenerate a gimbal control instruction. The smart glasses may becontrolled to further send the gimbal control instruction directly tothe UAV through the first channel.

That is, the attitude of the smart glasses can be sensed by the attitudesensor of the smart glasses and converted into a corresponding gimbalcontrol instruction for controlling the gimbal at the UAV and an imagingapparatus carried by the gimbal, such as a camera, a camera telescope, aremote camera, or a measuring instrument. In some embodiments, thegimbal control instruction may be used to achieve stabilization of theimaging apparatus, to adjust an attitude of the imaging apparatus, e.g.,a tilt angle and a photographing direction of the imaging apparatus, forachieving high-quality shooting and/or photographing, etc. For example,a camera of the imaging apparatus carried by the gimbal may be adjusteddownward by adjusting the smart glasses downward, and the camera of theimaging apparatus carried by the gimbal may be adjusted to the left byturning the smart glasses to the left. The examples are for illustrativepurposes and are not intended to limit the scope of the presentdisclosure.

In some embodiments, the smart glasses may also be controlled to sense auser gesture and generate a glasses control instruction, and to performa corresponding operation according to the glasses control instruction.The glasses control instruction may include an instruction for settingthe smart glasses, such as display screen backlight brightnessadjustment, a resolution of the smart glasses, or the like, and may alsoinclude a control instruction for controlling the imaging apparatus ofthe UAV, such as camera settings, photographing, video recording, etc.

That is, the menu operation of the smart glasses can be replaced by thesmart glasses sensing user gestures, and smart glasses menu functionscan be realized through gesture operations. In some embodiments, thesmart glasses may include a front camera. In an augmented reality mode,an operation may be performed by the user using a finger to click on aposition where a glasses icon is located. For example, in an augmentedreality mode, operation menus of the smart glasses may be displayed onan image plane. The user may click on a corresponding operation menu.The camera of the smart glasses may capture an image of the user's clickoperation and identify coordinates of a stable position of the user'sfinger in a certain time duration. By comparing the stable position ofthe user's finger with positions of the menus in the glasses, a glassescontrol instruction corresponding to the user's gesture operation may berecognized and a corresponding operation may be performed.

In some embodiments, the control method may further include controllingthe mobile terminal to establishes a second channel directly with thesmart glasses, and controlling the smart glasses to further send FPVimage data and/or flight status data received from a UAV to the mobileterminal through the second channel for display.

In some embodiments, the mobile terminal may be controlled to furthergenerate a first flight control instruction and/or a gimbal controlinstruction according to user input, and to send the first flightcontrol instruction and/or the gimbal control instruction to the smartglasses through the second channel. Further, the smart glasses may sendthe first flight control instruction and/or the gimbal controlinstruction to the UAV through the first channel.

In some embodiments, the control method may further include controllingthe mobile terminal to establish a wireless channel directly with theUAV, controlling the mobile terminal to receive FPV images from the UAVthrough the wireless channel and display the FPV images, and controllingthe UAV to receive a flight control instruction from the mobile terminalthrough the wireless channel.

That is, the UAV can communicate with the smart glasses through thewireless channel, to perform image transmission and to send and receivecontrol signals. In some other embodiments, the UAV can communicate withthe mobile terminal through wireless WiFi, to perform image transmissionand to send and receive control signals.

In some embodiments, a plurality of smart glasses may be added to theUAV control system to operate together as needed.

In some embodiments, the smart glasses may also be controlled to obtaina video file of the UAV directly from the UAV through a wired orwireless channel, and play back the video file on the smart glasses.

In the above-describe method, the smart glasses and the mobile terminalmay be connected through a wireless connection and/or a wiredconnection, e.g., a USB connection.

The present disclosure provides a control method and a control systemfor FPV UAV flight. The control system may include a remote controllerand smart glasses. the smart glasses can establish a first channeldirectly with the UAV without relaying by the remote controller, suchthat the smart glasses can receive FPV image data directly from the UAVthrough the first channel and display the FPV image data. The remotecontroller can establish a second channel directly with the UAV withoutrelaying by the smart glasses, such that the remote controller candirectly send a first flight control instruction to the UAV through thesecond channel. Accordingly, the UAV may be directly controlled withouta need for relaying, delay during transmission of images control signalsmay be reduced, and a control accuracy may be improved.

Those of ordinary skill in the art will appreciate that the exemplaryelements and algorithm steps described above can be implemented inelectronic hardware, or in a combination of computer software andelectronic hardware. Whether these functions are implemented in hardwareor software depends on the specific application and design constraintsof the technical solution. One of ordinary skill in the art can usedifferent methods to implement the described functions for differentapplication scenarios, but such implementations should not be consideredas beyond the scope of the present disclosure.

For simplification purposes, detailed descriptions of the operations ofexemplary systems, devices, and units may be omitted and references canbe made to the descriptions of the exemplary methods.

The disclosed systems, apparatuses, and methods may be implemented inother manners not described here. For example, the devices describedabove are merely illustrative. For example, the division of units mayonly be a logical function division, and there may be other ways ofdividing the units. For example, multiple units or components may becombined or may be integrated into another system, or some features maybe ignored, or not executed. Further, the coupling or direct coupling orcommunication connection shown or discussed may include a directconnection or an indirect connection or communication connection throughone or more interfaces, devices, or units, which may be electrical,mechanical, or in other form.

The units described as separate components may or may not be physicallyseparate, and a component shown as a unit may or may not be a physicalunit. That is, the units may be located in one place or may bedistributed over a plurality of network elements. Some or all of thecomponents may be selected according to the actual needs to achieve theobject of the present disclosure.

In addition, the functional units in the various embodiments of thepresent disclosure may be integrated in one processing unit, or eachunit may be an individual physically unit, or two or more units may beintegrated in one unit.

A method consistent with the disclosure can be implemented in the formof computer program stored in a non-transitory computer-readable storagemedium, which can be sold or used as a standalone product. The computerprogram can include instructions that enable a computer device, such asa personal computer, a server, or a network device, to perform part orall of a method consistent with the disclosure, such as one of theexemplary methods described above. The storage medium can be any mediumthat can store program codes, for example, a USB disk, a mobile harddisk, a read-only memory (ROM), a random access memory (RAM), a magneticdisk, or an optical disk.

Other embodiments of the disclosure will be apparent to those skilled inthe art from consideration of the specification and practice of theembodiments disclosed herein. It is intended that the specification andexamples be considered as exemplary only and not to limit the scope ofthe disclosure, with a true scope and spirit of the invention beingindicated by the following claims.

What is claimed is:
 1. A system for controlling an unmanned aerialvehicle (UAV) comprising: smart glasses configured to establish a firstchannel directly with the UAV, receive first person view (FPV) imagedata directly from the UAV through the first channel, and display theFPV image data; a remote control configured to establish a secondchannel directly with the UAV and send a first flight controlinstruction corresponding to the FPV image data directly to the UAVthrough the second channel, and establish a fourth channel directly withthe mobile terminal; and a mobile terminal configured to display the FPVimage data received from the UAV and forwarded by the remote control,wherein: the smart glasses are further configured to establish a thirdchannel directly with the remote control for the remote control to send,through the third channel, at least one of a disabling instruction fordisabling one or more menu functions of the smart glasses, a first smartglasses control instruction for adjusting a display parameter of thesmart glasses, or a second smart glasses control instruction forcontrolling a camera setting menu for the smart glasses, to the smartglasses, wherein a gimbal configured on the UAV carries a first camerafor collecting the FPV image data, the disabled one or more menufunctions correspond to a second flight control instruction forcontrolling flight of the UAV or a gimble control instruction forcontrolling a rotation of the gimbal, and the camera setting menu isconfigured to set the first camera.
 2. The system according to claim 1,wherein the smart glasses are further configured to receive flightstatus data directly from the UAV through the first channel and displaythe flight status data.
 3. The system according to claim 2, wherein thesmart glasses are further configured to display the flight status databy superimposing the flight status data on the FPV image data in anaugmented reality mode.
 4. The system according to claim 1, wherein thesmart glasses are further configured to: sense an attitude of the smartglasses; generate the gimbal control instruction according to theattitude; and send the gimbal control instruction directly to the UAVthrough the first channel.
 5. The system according to claim 1, whereinthe smart glasses are further configured to: sense a user gesture;generate a glasses control instruction according to the user gesture;and perform an operation according to the glasses control instruction.6. The system according to claim 1, wherein: the smart glasses arefurther configured to: sense a user gesture; generate the second flightcontrol instruction according to the user gesture; and send the secondflight control instruction directly to the UAV through the firstchannel.
 7. The system according to claim 6, wherein the smart glassesinclude: a second camera configured to capture an image of a usergesture operation; and a processor configured to: process the image ofthe user gesture operation; determine coordinates of the user gestureoperation; and compare the coordinates of the user gesture operationwith an imaging position of a virtual operation interface displayed onthe smart glasses for controlling the UAV, to recognize the secondflight control instruction corresponding to the user gesture operationand to perform an operation corresponding to the second flight controlinstruction.
 8. The system according to claim 6, wherein the smartglasses include: a second camera configured to capture a user gesturemotion; a memory for storing a plurality of user gesture motion modelsand a plurality of second flight control instructions corresponding tothe plurality of user gesture motion models; and a processorelectrically coupled to the second camera and the memory, and configuredto: generate a gesture motion model according to the user gesturemotion; compare the generated gesture motion model with the plurality ofuser gesture motion models stored in the memory; and trigger the secondflight control instruction corresponding to the generated gesture motionmodel in response to the generated gesture motion model being consistentwith one of the plurality of user gesture motion models stored in thememory.
 9. The system according to claim 8, wherein the plurality ofuser gesture motion models stored in the memory are obtained byphotographing a plurality of user gesture motions through the secondcamera and defining the user gesture motion models to correspond to theplurality of user gesture motions, respectively.
 10. The systemaccording to claim 1, wherein the remote control comprises: buttonsconfigured to implement the menu function of the smart glasses, whereinthe buttons are five-dimensional buttons, and wherein the menu functionincludes display brightness, camera settings, photographing, and videorecording.
 11. The system according to claim 1, wherein the remotecontrol is further configured to: receive at least one of the FPV imagedata or flight status data from the UAV through the second channel. 12.The system according to claim 1, wherein: the mobile terminal isconfigured to generate at least one of a third flight controlinstruction or a gimbal control instruction according to user input andsend the at least one of the third flight control instruction or thegimbal control instruction to the remote control through the fourthchannel; and the remote control is further configured to send the atleast one of the first flight control instruction or the gimbal controlinstruction to the UAV through the second channel.
 13. The systemaccording to claim 1, wherein the smart glasses are further configuredto send flight status data received from the UAV to the remote controlthrough the third channel for forwarding to the mobile terminal throughthe fourth channel for display.
 14. The system according to claim 13,wherein: the mobile terminal is configured to generate at least one of athird flight control instruction or a gimbal control instructionaccording to user input and send the at least one of the third flightcontrol instruction or the gimbal control instruction to the smartglasses through the remote control; and the smart glasses are furtherconfigured to send the at least one of the second flight controlinstruction or the gimbal control instruction to the UAV through thefirst channel.
 15. The system according to claim 1, wherein the mobileterminal is further configured to: establish a wireless channel directlywith the UAV; receive the FPV image data from the UAV through thewireless channel and display the FPV image data; and send a third flightcontrol instruction according to user input to the UAV through thewireless channel.
 16. The system according to claim 1, wherein the smartglasses are further configured to: obtain a video file directly from theUAV through the first channel; and play back the video file.
 17. Thesystem according to claim 1, wherein the display parameter of the smartglasses includes a display backlight brightness of the smart glasses.18. A method for controlling an unmanned aerial vehicle (UAV)comprising: controlling smart glasses to establish a first channeldirectly with the UAV; controlling the smart glasses to receive firstperson view (FPV) image data directly from the UAV through the firstchannel and to display the FPV image data; controlling a remote controlthat is separate from the smart glasses to establish a second channeldirectly with the UAV and send a first flight control instructioncorresponding to the FPV image data directly to the UAV through thesecond channel; controlling a mobile terminal to display the FPV imagedata received from the UAV and forwarded by the smart glasses;controlling the smart glasses to establish a third channel directly withthe remote control, and controlling the remote control to send, throughthe third channel, at least one of a disabling instruction for disablingone or more menu functions of the smart glasses, a first smart glassescontrol instruction for adjusting a display parameter of the smartglasses, or a second smart glasses control instruction for controlling acamera setting menu for the smart glasses, to the smart glasses; andcontrolling the remote control to establish a fourth channel directlywith the mobile terminal, wherein a gimbal configured on the UAV carriesa first camera for collecting the FPV image data, the disabled one ormore menu functions correspond to a second flight control instructionfor controlling flight of the UAV or a gimbal control instruction forcontrolling a rotation of the gimbal, and the camera setting menu isconfigured to set the first camera.
 19. The method according to claim18, further comprising: controlling the smart glasses to sense a usergesture to generate the second flight control instruction by:controlling a second camera disposed on the smart glasses to capture animage of a user gesture operation, processing the image of the gestureoperation to determine coordinates of the user gesture operation, andcomparing the coordinates of the user gesture operation with an imagingposition of a virtual operation interface displayed on the smart glassesfor controlling the UAV, to recognize the second flight controlinstruction corresponding to the user gesture operation and to performan operation corresponding to the second flight control instruction; orcontrolling the second camera to capture a user gesture motion,generating a gesture motion model according to the user gesture motion,comparing the gesture motion model with a plurality of pre-stored usergesture motion models in the smart glasses, and, in response to thegesture motion model being consistent with one of the plurality ofpre-stored user gesture motion models, triggering the second flightcontrol instruction corresponding to the gesture motion model; andcontrolling the smart glasses to send the second flight controlinstruction directly to the UAV through the first channel.